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INTRODUCTION 


The assessment study is a form of planning research that seeks 
to anticipate the secondary social impacts that might arise from: 

(1) The application from some new technology. 

(2) Government or private programs to cope with a major social 
problem like poverty, environmental pollution, or public 
safety. 

(3) A concerted national effort to achieve a widely supported 
specific goal like landing a man on the moon or finding a 
cure for cancer.* 

From the point of view of methodology, I see nothing to gain from 
distinguishing among the three types of studies identified above. The 
process of tracing secondary reactions is pretty much the same whether 
the initiating force is the application of a new technology (e.g., 
two-way Cable TV, genetics engineering, or a revolution in food pro- 
duction methods) or an innovative social program (a major change in 
taxation, a national health insurance program, or a "landmark" 

Supreme Court decision relative to civil rights). 

A key task in any assessment study is the generation of social 
impact scenarios that seek to trace in some structured fashion the 
interactions among various social forces. However, before an analyst 
is ready to generate such scenarios, he must first address three pre- 
paratory tasks that are common to all assessment studies, and, in 
fact, to most paper-and-pencil public policy research. These pre- 
paratory tasks are: 


A recent MITRE paper speculates about both the potentialities and 
complexities that might be associated with an intensive effort to 
dramatically increase longevity in the United States. See: Social 

Priorities - The Dilemma of Quality Versus Quantity (Martin V. Jones 
- MITRE MTP-364) , December 1971. 


1 



(1) IDENTIFY RELEVANT QUESTIONS. The analyst's first task is 
to identify and make explicit a whole host of heterogeneous 
essentially unstructured questions that must be answered 
relative to the nature of the technology or problem being 
assessed, and to exogenous forces that are related to the 
technology or problem. 

(2) SYSTEMATICALLY STRUCTURE QUESTIONS. The second task is to 
arrange those questions systematically so that they can be 
a basis for hypothesizing cause-effect, problem-solution, 
action-consequence relationships . 

(3) COLLECT DATA. The analyst's ability to draw inferences, 
however, depends upon his ability to develop answers to 

the specific questions that he has identified and structured 
in the first and second steps. This means that he must 

collect data that will guide his intuitive judgements in 

/ 

deriving these answers. 

Before proceeding further, it should be noted that the notion of 
assessment studies is not new. For years, disciplinary research has 
produced assessment studies. Economists have made assessments of the 
impacts of new legislation (e.g., tax measures) on the national income 
level, market researchers have assessed the impacts of new products 
on a company's sales, sociologists have assessed the impact of a 
proposed change in the parole system on the crime rate, educators 
have assessed the impacts of a major curriculum innovation on student 
achievement, etc. 

Similarly, interdisciplinary analyses in recent years have 
"assessed" the comparative merits and shortcomings of, alternative 
courses of action for solving or alleviating specific problems. In 
the category of this interdisciplinary research there has been opera- 
tions research, cost-benefit analysis, cost-effectiveness analysis, 
systems analysis, management science, computer simulation, the Program 
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Evaluation and Review Technique, the Program-Planning-Budgeting 
system, and the so-called "Policy Sciences." 

However, a major characteristic of most of this disciplinary 
and interdisciplinary research is that it has sought answers to a 
relatively narrow list of questions. Economists have usually con- 
fined their efforts to appraising the impact of a particular measure 
on the nation's economic well being, market researchers have primarily 
been concerned with the effect that a new product would have on a parti- 
cular company ' s or industry* s sales or profit position. By the same 
token, most interdisciplinary studies have compressed the entire 
decision-making criterion into some simple cost-performance ratio, 
i.e., the dollar cost per patient serviced in a medical treatment 
center. 

One way of describing the contribution of the technology assess- 
ment movement is to refer back to the first of the three analytical 
tasks listed at the beginning of this paper. Those who have pioneered 
the technology assessment movement have insisted that the analyst 
must vastly increase the scope and the number of questions to which 
he seeks to develop answers. This point has been succinctly stated 
by Professor Mayo: 

Perhaps the most significant aspect of the concept of tech- 
nology assessment is that it is, and is meant to be, 
consistent with the notion of Total Impact Assessments , 
i.e., the identification of all social impacts of a parti- 
cular application rather than selected impacts. * 


Louis H. Mayo, Scientific Method, Adversarial System, and Tech- 
nology Assessment , November 1970, Program of Policy Studies in 
Science and Technology, George Washington University Monograph 
No. 5, p. 3. 
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THE MITRE-OST PROJECT 


I believe that our recent MITRE methodological studies for OST 
made a first step toward addressing in a generic context the first 
two tasks listed above. First, we tried to suggest, as comprehen- 
sively and as explicitly as time would allow, how the concept "total 
impact analysis" might be defined. In defining "relevant considera- 
tions" we consolidated lists of highly diverse societal characteristics 
in a somewhat different way than, to our knowledge, has therefore been 
done in either disciplinary or interdisciplinary studies. In so doing 
we drew extensively from the published research of others in many 
fields for the component items of our lists. These lists of societal 
characteristics - covering such matters as values and goals, demo- 
graphy, environment, economic factors, social elements, and insti- 
tutional parameters - provided a beginning master list of areas of 
interest about which the analyst should raise questions when he begins 
the process of making a total impact assessment study. 

In the MITRE-OST study we also tried to contribute in a generic 
way to the second task identified at the beginning of the paper. We 
provided a seven-step procedure which, we believe, can help an 
analyst to integrate the diverse checklists of questions so that he 
can begin to trace in a comprehensive fashion the initial and secondary 
impacts of any major technological application or of society's attempts 
to respond to or redirect that application. Exhibit 1 provides an 
analytical overview of the seven-step procedure and some of the 
supporting checklists. 
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EXHIBIT 1 

TECHNOLOGY ASSESSMENT: 

A METHODOLOGICAL OVERVIEW 


















































ILLUSTRATIVE SOCIAL- IMP ACT SCENARIOS 


During the last year our thinking at MITRE has moved toward 
increasingly explicit social impact scenarios. Most of the scenarios 
in the study for OST were essentially simple and qualitative. For 
instance, Exhibit 2 lists in a relatively straightforward manner a 
partial series of historical events following the introduction of 
man-made fabrics. Exhibit 3 uses a flow-diagram technique to depict 
some multidimensional impacts that might follow an accelerated auto- 
mation in industry. Exhibit 4 in a similar way depicts some conse- 
quences that might ensue if mariculture (sea-farming) were successfully 
applied to reduce malnutrition in developing countries. 

In Exhibit 5 we speculate about one set of consequences that 
might follow the introduction of two-way Cable TV in major cities. 

This exhibit elaborates the scenario process by documenting the 
rationale that led us to hypothesize the series of events shown. 

In Exhibit 6 we carry the methodology substantially further in 

that: 

(1) we attach four important qualifying and elaborating bits of 
information to each successive event: 

(a) how probable is it that the interaction will, in fact, 
occur? 

(b) in what direction will the interaction occur, i.e., 
will the happening of the earlier event cause the 
later event to increase or decrease? 

(c) what will likely be the magnitude of the interaction 
if it occurs? 

(d) what will be the timing of the interaction? How long 
after the earlier event will the later event occur? 

(2) we show multiple consequences flowing from one prior event 
rather than a single consequence. 


6 



EXHIBIT 2 


SELECTED IMPACTS OF THE INTRODUCTION OF 
MAN-MADE FABRICS ON CITIES 


less use of cotton in clothing manufacture 

decline in sales of U. S. grown cotton 

reduced employment opportunities for unskilled 
blacks in southern cotton fields 


stimulated migration of southern blacks to 
northern cities 


great expansion of welfare costs in northern 
cities 


financial crises in northern cities involving 
huge increases in city obligations without com- 
mensurate increases in the tax base and revenues 


steady exodus of northern urban whites to 
suburbs 


increasing political influence of non- 
whites in northern cities 


election of black officials in northern cities 



EXHIBIT 3 


SOME POSSIBLE CONSEQUENCES 
OF A RAPID INCREASE IN INDUSTRIAL AUTOMATION 


1 and 8 
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IMPACTS SUBSEQUENT TO THE REDUCTION OF MALNUTRITION FOLLOWING MARICULTURE 

APPLICATION IN DEVELOPING COUNTRIES 
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POSSIBLE METHODOLOGICAL REFINEMENTS 


There is, as I see it, a distinct advantage to the increasingly 
explicit scenarios as one moves from Exhibit 2 through 6. A scenario, 
like Exhibit 6, is much more informative as to exactly what conditions 
an analyst is, in fact, projecting. Traditionally, a shortcoming of 
many scenarios and projections is their non-explicitness. Because it 
is both explicit and discrete, a scenario, like Exhibit 6, makes it 
possible for other analysts to concur with, or take exception to, 
specific entries in the scenario without having to accept or reject 
the scenario totally. 

Although Exhibit 6 is much more informative than Exhibit 2 or 3, 
Exhibit 6 is also simplified as to the scope of information that 
should ideally be shown in this type scenario. For instance: 

(1) Other factors that would either reinforce or dampen the 
specified sequential relationships should be included in 
the scenario. For instance, in Exhibit 6 disbursements 
from the Civil Service Retirement Fund are likely to be 
influenced by many other factors besides changes in the 
mandatory retirement age - e.g., changes in the number of 
persons reaching 65 years of age, changes in the price 
level, etc. 

(2) Exhibit 6 shows only 11 interactions. A scenario that 
truly aimed to model the real world might require 50, 100, 
or more interactions. For instance, if the movement of 
the aged to Florida and the Far West were slowed, this 
would lead to (+) impacts on the economic prosperity and 
the political power of northern states vs. Florida and 
the Far West. Similarly, lower-level impacts might be 
anticipated - the demand for winter clothing would be 
increased and that for golf and fishing equipment reduced. 
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(3) Exhibit 6 shows only one-way interactions. An indepth 
scenario would aim to include dual or two-way interactions, 
e.g., not only do economic events influence demography, 
but demographic events cause economic impacts. 

(4) Obviously, in any given case, what is required is not a 
single scenario, but a whole series of them. The qualify- 
ing coefficients to the events specified in Exhibit 6 are 
all single-valued. However, each of the matters covered 
is characterized by uncertainty. An alternative scenario 
is needed to trace the chain of consequences if a given 
probability of occurrence were to change from, say, 0.9 to 
0.5, or, if, the magnitude of change were to be weak 
instead of strong, or the timing 25-36 months instead of 
1-12 months. 

(5) The qualifying descriptive bits of information attached to 

each sequential event should be expanded beyond the four 
shown in Exhibit 6 - probability of occurrence, direction, 
magnitude, and timing of impacts. Other information that 
might be shown includes: the duration of the impact, the 

diffusion of it across society totally or among members of 
a specified target group,* and the estimated extent to 
which the impact may be amenable to social control. 

Research in developing more explicit, sophisticated scenarios 
should be accompanied by parametric empirical research that would 


For instance, the "aged" are not a monolithic group. Some are rich, 
others are poor; some are well, others are sick; some desire to work, 
others do not; some are highly trained, others are unskilled; some 
are married, others are not; some live in cities, others on farms, 
etc. The specific impact that a new technology or social program 
would have would vary greatly according to the socio-economic con- 
dition of the aged person involved. 
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help an analyst estimate the probability, direction, magnitude, 
timing, etc. of the various entries in the scenarios. For instance, 
a major initial concern is whether the period of application of a 
new technology will be quick and short or slow and protracted. 

Exhibit 7 lists some of the factors that can influence the length 
of this application period. Similar parametric lists should be 
developed that would help an analyst to estimate the probability, 
direction, magnitude, and timing of the secondary consequences that 
follow from the initial application of the innovation. 

Going beyond the qualitative parametric relationships just cited, 
the next step is to quantify the relationships, wherever possible. 

The MITRE reports for OST cited, illustratively, a wide variety of 
such quantitative relationships that have been developed in many 
fields — economics, demography, environment, public safety, health, 
etc. (See: Volume 1, pp. 87-92). As one specific example, Exhibit 8 

extracts a small portion of a computer analysis that was conducted for 
the MITRE mariculture pilot assessment study. This analysis projects 
quantitatively the potential impacts on 26 different socio-economic- 
environmental conditions of mariculture applications in 67 developing 
countries . 
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EXHIBIT 7 


FACTORS THAT WILL INFLUENCE HOW LONG IT WILL 
TAKE TO APPLY A NEW TECHNOLOGY 


_ FACTORS EXPLANATION 

People vs. Things If a new technology initially impacts on the material 

world, such as the transistor did, there will probably 
be less delay in its widespread adoption than if it 
impacts in a major way on people physiologically. 

There almost surely will be a delay if the product is 
one that people would ingest, as a new powerful drug, 
that might have serious adverse side effects. 

Nature of Decision Making Centralized decision making, such as in the military or 

space programs, is conducive to more rapid application 
than diffused decision making involving many checks 
and balances, as is currently the situation in certain 
new health technologies. 

National Commitment If the new technology would satisfy a "crying need" (a 

cure for cancer) or a national goal (to land a man on 
the moon), there will normally be a tendency to assume 
risks or surmount obstacles that would otherwise block 
or delay an application. 

Reward for Innovator Since most innovations in our private enterprise 

economy are made by entrepreneurs, how the rate of 
application affects entrepreneurial profits is import- 
ant. Sometimes, for various reasons, it has been in 
the interest of the innovator (e.g., Corfam, substi- 
tute for leather) to prolong the application period. 

In other cases, where imitation has been easy and pro- 
duct differentiation difficult (as in the fashion 
field), there has been a tendency to exploit the market 
quickly . 

Capital Required All other things being equal, the larger the capital 

investment required, the more restricted the number of 
organizations that can participate in the application, 
and hence, the slower the rate of application. The 
increasing capital investment required for the develop- 
ment of birth control devices is one of the reasons 
that the period of application of new technology in 
this field may lengthen. 

Competition Closely linked to several prior considerations is the 

extent of competition in both research and production. 
In many industries, smaller companies whose fortunes 
in the industry are rising set the pace for rapid 
application of new technology. In other industries 
where the industry structure is stabilized or mori- 
bund, innovation is slow. 

Institutional Climate Again, similar to several of the above, the extent to 

which vested interests can conspire to stymie innova- 
tion will greatly influence the rate at which innova- 
tion is applied. The building industry is, of course, 
the classic case where contractors, labor unions, and 
local building codes have for all practical purposes 
throttled major innovations. 
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SOME ANTICIPATED IMPACTS RESULTING FROM MARICULTURE APPLICATION IN 67 DEVELOPING COUNTRIES* 
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DATA COLLECTION 


In the first and second tasks leading up to the social-impact 
scenarios that we have discussed thus far, we illustrated some of 
the social characteristics that should be related in an assessment 
study. We also identified, again illustratively, some of the para- 
meters - like probability of occurrence, direction of change, 
magnitude of change, and timing of occurrence - that should be traced 
for each of the interrelated factors. The third task is to collect 
data that will make it possible to assign the coefficients to these 
parameters in any given case. Should the probability of occurrence 
of one event following another be designated 0.1, 0.5, or 0.9? 

Should the timing be placed in the 0-12 months range, 13-24 months, 
or over three years? 

The MITRE study for OST, primarily because of time limitations, 
did not explore this issue of data collection to the same extent as 
it did the first two tasks, identifying the questions to be addressed 
and structuring these questions systematically for analytical purposes. 
Actually the task of collecting data for a technological assessment 
study is not essentially different than that of any other future- 
oriented, public-policy-issue, paper-and-pencil study. Probably the 
major difference, as noted previously, is that in an assessment study 
information would have to be collected on a much wider variety of 
matters — values, demography, economics, environment, social issues, 
and institutional considerations — than in a typical disciplinary 
or even interdisciplinary cost/benefit study. For some of these 
matters - like values and institutional considerations - it is also 
more difficult to collect "hard data" than it is in the typical 
economic research or market analysis survey. 

However, rarely, for any of these matters is the choice one of 
data vs. no data at all. It is rather one of data of various shades 
of relevance and validity. It may also be a question of documented 
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data vs. undocumented ("expert opinion," "authorative source") data. 
Other things being equal, documented data are preferred to undocu- 
mented data because it is normally easier to doublecheck and verify 
documented data. However, often other things are not equal. For 
instance, in dealing with new somewhat unique projects, undocumented 
expert opinion data may sometimes be just as good or better than 
documented data because the so-called undocumented data are more 
current and relevant. For instance, a carefully developed "guessti- 
mate" from a well-known gerontologist might provide a sounder 
estimating base relative to the effect on senior citizen productivity 
and morale of raising the compulsory retirement age in the United 
States than would a written report prepared at an earlier date in a 
different country with a somewhat different cultural heritage. In 
recent years new methods have been developed for systematically 
reaching a consensus of expert opinion on a given subject, including 
future forecasts. The best known of these methods is the Delphi 
Technique. 

In searching for data, the assessment analyst should make use 
of all of the analytical techniques that economic, technological, and 
other forecasters have been using for years. There is no point to 
discuss these techniques in detail here. The MITRE study for OST 
(Volume 1, Chapter XII) has a brief chapter on forecasting, and, of 
course, the literature abounds with long books on the subject. As a 
source of possible interest relative to my own views on forecasting 
methods, I have reproduced in Exhibit 9 a one-page recap of fore- 
casting methods that appeared in the referenced chapter. 

In the realm of documented data, the conventional planning factor 
which expresses the quantitative historical relationship between one 
type of event and another is certainly a useful forecasting tool for 
the assessment analyst in tracing both the timing and magnitude of 
societal interactions. Economists, of course, have a large inventory 
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EXHIBIT 9 


A RECAP OF FORECASTING METHODS 
(Hypothetical Question: What Percentage 
of U.S. Physicians Will Use Computer 
Diagnostic Services by 1985?) 


FORECASTING METHODS 


DEFINITION 


EXAMPLE 


INTUITION 

A forecast baaed on the 
subjective Judgment of the 
forecaster . 


Experts at an extemporaneous workshop session of a 
joint physician, computer-industry symposium predict 
that by 1985 approximately 65% of U. S. physicians 
will employ computer diagnostic services. They cite 
as evidence the increasing experimentation with the 
use of automated techniques in the medical 
profession. 


TREND EXTRAPOLATION 


A forecast baaed on the 
assumption of the continua- 
tion into the future of some 
discerned past trend. 


Statistics show that over the past 15 years the per- 
centage of physicians using computer diagnostic 
services increased frcm 4 to 27%. Continuing that 
trend for the next 15 years Indicates that by 1985 
approximately 65% of physicians will employ computer 
diagnostic services* 


TREND CORRELATION 

A forecast of the future 
status of some phenomenon in 
terms of a consistent rela- 
tionship of that phenomenon 
to same other phenomenon in 
the past whose future status 
has already been projected. 


Historical data covering the last 10 years show that 
the percentage of physicians with access to computer 
diagnostic consultation is well correlated with 
three other factors: the Increase in private group 

medical practice, the percentage of the population 
covered by medical insurance, and the percentage of 
doctors graduated from medical schools offering in- 
struction in medical applications of computers. 
Projections on these three factors are available 
through 1985. Using these projections as a basis, a 
statistical correlation analysis indicates that by 
1985 65% of physicians will have access to computer 
diagnostic consultation. 


MODELS (STATISTICAL) 

This method la a much elab- 
orated version of the his- 
torical trend correlation 
technique described above. 

It often involves the use of 
dozens, and sometimes of 
hundreds, of estimating 
equations — all Integrated 
into a unified forecasting 
method. 


An in depth study of physicians who have already 
adopted computer diagnostic consultation services 
shows that such usage is related in a complex way to 
some 10 different variables such as physician work 
load, degree of medical specialization, the access 
to and use of other consultative serrlees, the cost 
of the computer service, etc. Well documented 
studies make it possible to predict the growth 
factor through 1985 for these 10 governing variables. 
Using this later study and the cited historical re- 
lationship, it is possible to predict that 65% of 
physicians will employ computer diagnostic consul- 
tation in 1985. 


ANALOGY 

This method predicts the 
future by drawing a plaus- 
ible parallel between the 
future and some presumably 
similar prior event. 


In terms of many management and scientific services 
the medical research field has been about 25 years 
ahead of the practicing physician. In 1960 approxi- 
mately 65% of the nation's medical research facili- 
ties were using computers for data analysis and 
synthesis tasks similar to those involved in physi- 
cian computer diagnostic consultation. On this 
basis it is predicted that by 1985 approximately 
65% of physicians will employ computer diagnostic 
consultation services. 
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of such relationships that are expressed in "multiplier" and 
"acceleration" principles. Usually these factors express the quanti- 
tative relationships between investment, production, employment, 
income, spending, etc. There are also temporal relationships involv- 
ing market and social behavior. For instance, changes in wholesale 
prices usually precede changes in retail prices by several months. 
Demographers, environmentalists, sociologists, medical technicians, 
traffic engineers, and other specialists have similar inventories of 
rule-of-thumb planning relationships. 

The appropriate caveats applying to such relationships are well 
known. All such relationships are developed from historical (hopefully 
analogous) experience. Since we can say for sure that the future will 
seldom be a carbon copy of the past, at best, such historical relation- 
ships can be taken only as approximate guides to future relationships. 
Also, in most cases these quantitative relationships only describe 
the past in highly gross terms. In spite of the arithmetic precision 
with which these relationships are often expressed, they usually are 
simple averages that conceal much variation, and sometimes experts 
even disagree as to what the average historical relationships are. 
Notwithstanding these caveats, the assessment analyst must use these 
planning factors. If he discards them completely, he is left with 
nothing but unadulterated intuition and heresay, and normally he 
has no sound basis for selecting one unsupported intuitive judgement 
over another. 

In the months ahead, we at MITRE hope to explore further the 
possibilities of new methods of data generation for making assessment 
studies. 
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